{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data exploration - SQuAD v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Common imports \n",
    "import pandas as pd\n",
    "from IPython.display import Markdown, display, clear_output\n",
    "from nltk import tokenize\n",
    "from scipy import stats\n",
    "from IPython.core.debugger import set_trace\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretty printing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def printBold(string):\n",
    "    display(Markdown('**' + string + '**'))\n",
    "    \n",
    "#TODO    \n",
    "#def printColor():\n",
    "#     display(Markdown('<span style=\"color:blue\">blue</span>'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pickling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import _pickle as cPickle\n",
    "from pathlib import Path\n",
    "\n",
    "def dumpPickle(fileName, content):\n",
    "    pickleFile = open(fileName, 'wb')\n",
    "    cPickle.dump(content, pickleFile, -1)\n",
    "    pickleFile.close()\n",
    "\n",
    "def loadPickle(fileName):    \n",
    "    file = open(fileName, 'rb')\n",
    "    content = cPickle.load(file)\n",
    "    file.close()\n",
    "    \n",
    "    return content\n",
    "    \n",
    "def pickleExists(fileName):\n",
    "    file = Path(fileName)\n",
    "    \n",
    "    if file.is_file():\n",
    "        return True\n",
    "    \n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we aren't really doing the answering of the questions, as is the true intention for the dataset, we'll merge the train and dev datasets into one. The test dataset is probably hidden, since there's a competition for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_json('../data/squad-v1/train-v1.1.json', orient='column')\n",
    "dev = pd.read_json('../data/squad-v1/dev-v1.1.json', orient='column')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([train, dev], ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data</th>\n",
       "      <th>version</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'title': 'University_of_Notre_Dame', 'paragra...</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'title': 'Beyoncé', 'paragraphs': [{'context'...</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'title': 'Montana', 'paragraphs': [{'context'...</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'title': 'Genocide', 'paragraphs': [{'context...</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'title': 'Antibiotics', 'paragraphs': [{'cont...</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                data  version\n",
       "0  {'title': 'University_of_Notre_Dame', 'paragra...      1.1\n",
       "1  {'title': 'Beyoncé', 'paragraphs': [{'context'...      1.1\n",
       "2  {'title': 'Montana', 'paragraphs': [{'context'...      1.1\n",
       "3  {'title': 'Genocide', 'paragraphs': [{'context...      1.1\n",
       "4  {'title': 'Antibiotics', 'paragraphs': [{'cont...      1.1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a what we've got."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def showQuestion(titleId, paragraphId, questionId):\n",
    "\n",
    "    title = df['data'][titleId]['title']\n",
    "    paragraph = df['data'][titleId]['paragraphs'][paragraphId]['context']\n",
    "    question = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['question']\n",
    "    answer = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['text']\n",
    "    answerStart = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['answer_start']\n",
    "\n",
    "    printBold('Title')\n",
    "    print(title)\n",
    "    printBold('Paragraph')\n",
    "    print(paragraph)\n",
    "    printBold('Question')\n",
    "    print(question)\n",
    "    printBold('Answer')\n",
    "    print(answerStart)\n",
    "    print(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "University_of_Notre_Dame\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "515\n",
      "Saint Bernadette Soubirous\n"
     ]
    }
   ],
   "source": [
    "titleId = 0\n",
    "paragraphId = 0 \n",
    "questionId = 0\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Titles 490\n",
      "Paragraphs 20963\n",
      "Questions 98169\n"
     ]
    }
   ],
   "source": [
    "titlesCount = len(df['data'])\n",
    "totalParagraphsCount = 0\n",
    "totalQuestionsCount = 0\n",
    "\n",
    "for titleId in range(titlesCount):\n",
    "    paragraphsCount = len(df['data'][titleId]['paragraphs'])\n",
    "    totalParagraphsCount += paragraphsCount\n",
    "    \n",
    "    for paragraphId in range(paragraphsCount):\n",
    "        questionsCount = len(df['data'][titleId]['paragraphs'][paragraphId]['qas'])\n",
    "        \n",
    "        totalQuestionsCount += questionsCount\n",
    "        \n",
    "print('Titles', titlesCount)\n",
    "print('Paragraphs', totalParagraphsCount)\n",
    "print('Questions', totalQuestionsCount)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "University_of_Notre_Dame\n",
      "Beyoncé\n",
      "Montana\n",
      "Genocide\n",
      "Antibiotics\n",
      "Frédéric_Chopin\n",
      "Sino-Tibetan_relations_during_the_Ming_dynasty\n",
      "IPod\n",
      "The_Legend_of_Zelda:_Twilight_Princess\n",
      "Spectre_(2015_film)\n",
      "2008_Sichuan_earthquake\n",
      "New_York_City\n",
      "To_Kill_a_Mockingbird\n",
      "Solar_energy\n",
      "Tajikistan\n",
      "Anthropology\n",
      "Portugal\n",
      "Kanye_West\n",
      "Buddhism\n",
      "American_Idol\n"
     ]
    }
   ],
   "source": [
    "titles = []\n",
    "for titleId in range(len(df['data'])):\n",
    "    titles.append(df['data'][titleId]['title'])\n",
    "    \n",
    "for i in range(20):\n",
    "    print(titles[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Titles are pretty random. Seems to be a lot of locations like countries and cities but not nearly enough to afford splitting the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of our main assumptions is that the sentence that contains the answer could be turned into a question just by removing the answer from it. Let's see how much of that is true for the questions in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "University_of_Notre_Dame\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "515\n",
      "Saint Bernadette Soubirous\n"
     ]
    }
   ],
   "source": [
    "titleId = 0\n",
    "paragraphId = 0 \n",
    "questionId = 0\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extractSentence(paragraph, answerStart):\n",
    "    \n",
    "    sentences = tokenize.sent_tokenize(paragraph)\n",
    "    sentenceStart = 0\n",
    "    \n",
    "    for sentence in sentences:\n",
    "        if (sentenceStart + len(sentence) >= answerStart):\n",
    "            return sentence         \n",
    "        \n",
    "        sentenceStart += len(sentence) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858.\n"
     ]
    }
   ],
   "source": [
    "paragraph = df['data'][0]['paragraphs'][0]['context']\n",
    "answerStart = df['data'][0]['paragraphs'][0]['qas'][0]['answers'][0]['answer_start']\n",
    "\n",
    "sentence = extractSentence(paragraph, answerStart)\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def containedInText(text, question):\n",
    "    \n",
    "    questionWords = tokenize.word_tokenize(question.lower())\n",
    "    textWords = tokenize.word_tokenize(text.lower())\n",
    "    wordsContained = 0\n",
    "\n",
    "    for questionWord in questionWords:\n",
    "        for textWord in textWords:\n",
    "            if (questionWord == textWord):\n",
    "                wordsContained += 1\n",
    "                break\n",
    "\n",
    "    return wordsContained / len(questionWords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "question =  df['data'][0]['paragraphs'][0]['qas'][0]['question']\n",
    "\n",
    "contained = containedInText(sentence, question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Sentence**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Contained**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6428571428571429\n"
     ]
    }
   ],
   "source": [
    "printBold('Question')\n",
    "print(question)\n",
    "printBold('Sentence')\n",
    "print(sentence)\n",
    "printBold(\"Contained\")\n",
    "print(contained)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I wouldn't expect a 100% containment simply because the questions will contain **question-like words** like *Why, Who, *Whom*, What*.\n",
    "\n",
    "In this example we also see that the word appear is contained in the original sentence but in **past tense**. We could take care of that if we take the **stems** of the words, but I think it's better to see the least imaginative way for forming questions.\n",
    "\n",
    "We are also calculating some **stopwords - common words like *to, the, in*** which could be encountered at different places of the sentence, but again we want to measure the least-creative questions.\n",
    "\n",
    "In this sentece *(damn, that was a good example)* we also see that the question uses the word *allegedly* which is a **synonym** of *reputedly* in the sentence. That could be nice for question forming, but I think it's more of an overkill.\n",
    "\n",
    "We also see that the question actually encompasses the **words around the answer, rather than the entire sentence**. Which is a definate must-do when we form our questions. \n",
    "\n",
    "Let's see what is the score on all of the questons. I'm also curious to see the score on the entire paragraph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This may come in handy in the future. Pretty printing the progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Printint the percentage completed\n",
    "def printPercentage(currentStep, maxStep):\n",
    "    stepSize = maxStep / 100\n",
    "    \n",
    "    if (int(currentStep / stepSize) > ((currentStep - 1) / stepSize)):\n",
    "        clear_output()\n",
    "        print('{}%'.format(int(currentStep / stepSize)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pickle found. Saved some time.\n"
     ]
    }
   ],
   "source": [
    "questionContainmentDfPickleName = 'pickles/questionContainmentDf.pkl'\n",
    "\n",
    "#If the dataframe is already generated, load it.\n",
    "if (pickleExists(questionContainmentDfPickleName)):\n",
    "    print(\"Pickle found. Saved some time.\")\n",
    "    questionContainmentDf = loadPickle(questionContainmentDfPickleName)\n",
    "else:\n",
    "    sentenceScore = []\n",
    "    paragraphScore = []\n",
    "\n",
    "    #For each title\n",
    "    titlesCount = len(df['data'])\n",
    "    for titleId in range(titlesCount):\n",
    "        printPercentage(titleId, titlesCount)\n",
    "\n",
    "        #For each paragraph\n",
    "        for paragraphId in range(len(df['data'][titleId]['paragraphs'])):\n",
    "            paragraph = df['data'][titleId]['paragraphs'][paragraphId]['context']\n",
    "\n",
    "            #For each question\n",
    "            for questionId in range(len(df['data'][titleId]['paragraphs'][paragraphId]['qas'])):\n",
    "                question = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['question']\n",
    "                answerStart = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['answer_start']\n",
    "                sentence = extractSentence(paragraph, answerStart)\n",
    "\n",
    "                sentenceScore.append(containedInText(sentence, question))\n",
    "                paragraphScore.append(containedInText(paragraph, question))           \n",
    "                \n",
    "    #Merge dataframes into one                \n",
    "    sentenceScoreDf = pd.DataFrame(sentenceScore, columns=['sentence'])\n",
    "    paragraphScoreDf = pd.DataFrame(paragraphScore, columns=['paragraph'])\n",
    "\n",
    "    questionContainmentDf = pd.concat([sentenceScoreDf, paragraphScoreDf], axis=1)\n",
    "    \n",
    "    #Pickle the result\n",
    "    dumpPickle(questionContainmentDfPickleName, questionContainmentDf)\n",
    "    \n",
    "    print(\"Result not pickled. Generating...\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>paragraph</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>98169.000000</td>\n",
       "      <td>98169.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.463937</td>\n",
       "      <td>0.582157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.190377</td>\n",
       "      <td>0.159055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.461538</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           sentence     paragraph\n",
       "count  98169.000000  98169.000000\n",
       "mean       0.463937      0.582157\n",
       "std        0.190377      0.159055\n",
       "min        0.000000      0.000000\n",
       "25%        0.333333      0.500000\n",
       "50%        0.461538      0.600000\n",
       "75%        0.600000      0.700000\n",
       "max        1.000000      1.000000"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questionContainmentDf.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would argue that almost half the words contained is a pretty good result. \n",
    "\n",
    "As expected, contained within the entire paragraph is better.\n",
    "\n",
    "I do wonder about those questions that are 100% contained in the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>paragraph</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.642857</td>\n",
       "      <td>0.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.636364</td>\n",
       "      <td>0.636364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.375000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.636364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.363636</td>\n",
       "      <td>0.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.545455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.266667</td>\n",
       "      <td>0.733333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentence  paragraph\n",
       "0  0.642857   0.571429\n",
       "1  0.636364   0.636364\n",
       "2  0.533333   0.600000\n",
       "3  0.375000   0.500000\n",
       "4  0.333333   0.416667\n",
       "5  0.272727   0.636364\n",
       "6  0.300000   0.800000\n",
       "7  0.363636   0.727273\n",
       "8  0.000000   0.545455\n",
       "9  0.266667   0.733333"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questionContainmentDf.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getQuestionAt(index):\n",
    "    currentIndex = 0\n",
    "    \n",
    "    for titleId in range(len(df['data'])):\n",
    "        for paragraphId in range(len(df['data'][titleId]['paragraphs'])):\n",
    "            for questionId in range(len(df['data'][titleId]['paragraphs'][paragraphId]['qas'])):\n",
    "                if (currentIndex == index):\n",
    "                    return titleId, paragraphId, questionId\n",
    "                currentIndex += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see question #8 which has 0 containment in the answer sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1, 3)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getQuestionAt(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "University_of_Notre_Dame\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As at most other universities, Notre Dame's students run a number of news media outlets. The nine student-run outlets include three newspapers, both a radio and television station, and several magazines and journals. Begun as a one-page journal in September 1876, the Scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the United States. The other magazine, The Juggler, is released twice a year and focuses on student literature and artwork. The Dome yearbook is published annually. The newspapers have varying publication interests, with The Observer published daily and mainly reporting university and other news, and staffed by students from both Notre Dame and Saint Mary's College. Unlike Scholastic and The Dome, The Observer is an independent publication and does not have a faculty advisor or any editorial oversight from the University. In 1987, when some students believed that The Observer began to show a conservative bias, a liberal newspaper, Common Sense was published. Likewise, in 2003, when other students believed that the paper showed a liberal bias, the conservative paper Irish Rover went into production. Neither paper is published as often as The Observer; however, all three are distributed to all students. Finally, in Spring 2008 an undergraduate journal for political science research, Beyond Politics, made its debut.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How many student news papers are found at Notre Dame?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "126\n",
      "three\n"
     ]
    }
   ],
   "source": [
    "titleId = 0\n",
    "paragraphId = 1 \n",
    "questionId = 3\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The question is actually formed from the previous sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0% containment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>paragraph</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>363</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>505</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2781</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3678</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      sentence  paragraph\n",
       "269        0.0        0.0\n",
       "363        0.0        0.0\n",
       "505        0.0        0.0\n",
       "2781       0.0        0.0\n",
       "3678       0.0        0.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questionContainmentDf[questionContainmentDf['paragraph'] == 0].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 0, 0)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getQuestionAt(269)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beyoncé\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles \"Crazy in Love\" and \"Baby Boy\".\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When did Beyonce start becoming popular?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "269\n",
      "in the late 1990s\n"
     ]
    }
   ],
   "source": [
    "titleId = 1\n",
    "paragraphId = 0 \n",
    "questionId = 0\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **synonym** case - *instead of rose to fame*, *start becoming popular* is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 18, 6)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getQuestionAt(505)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beyoncé\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In 2011, documents obtained by WikiLeaks revealed that Beyoncé was one of many entertainers who performed for the family of Libyan ruler Muammar Gaddafi. Rolling Stone reported that the music industry was urging them to return the money they earned for the concerts; a spokesperson for Beyoncé later confirmed to The Huffington Post that she donated the money to the Clinton Bush Haiti Fund. Later that year she became the first solo female artist to headline the main Pyramid stage at the 2011 Glastonbury Festival in over twenty years, and was named the highest-paid performer in the world per minute.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When did this leak happen?\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "2011\n"
     ]
    }
   ],
   "source": [
    "titleId = 1\n",
    "paragraphId = 18 \n",
    "questionId = 6\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's just a bad question. It could only be asked in combination with the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 100% containment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>paragraph</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21911</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39394</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45064</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48874</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53226</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67425</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sentence  paragraph\n",
       "21911       1.0        1.0\n",
       "39394       1.0        1.0\n",
       "45064       1.0        1.0\n",
       "48874       1.0        1.0\n",
       "53226       1.0        1.0\n",
       "67425       1.0        1.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questionContainmentDf[questionContainmentDf['sentence'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(258, 23, 0)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getQuestionAt(53226)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Utrecht\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Utrecht city has an active cultural life, and in the Netherlands is second only to Amsterdam. There are several theatres and theatre companies. The 1941 main city theatre was built by Dudok. Besides theatres there is a large number of cinemas including three arthouse cinemas. Utrecht is host to the international Early Music Festival (Festival Oude Muziek, for music before 1800) and the Netherlands Film Festival. The city has an important classical music hall Vredenburg (1979 by Herman Hertzberger). Its acoustics are considered among the best of the 20th-century original music halls.[citation needed] The original Vredenburg music hall has been redeveloped as part of the larger station area redevelopment plan and in 2014 has gained additional halls that allowed its merger with the rock club Tivoli and the SJU jazzpodium. There are several other venues for music throughout the city. Young musicians are educated in the conservatory, a department of the Utrecht School of the Arts. There is a specialised museum of automatically playing musical instruments.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cultural life in Utrecht is second to \n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Utrecht city has an active cultural life, and in the Netherlands is second only to Amsterdam\n"
     ]
    }
   ],
   "source": [
    "titleId = 258\n",
    "paragraphId = 23 \n",
    "questionId = 0\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Strange question. The question words all appear in the sentence, but not in order. But the answer is the entire sentence, which obviously has needless information inside it. Looking further into it, the question is actually wrong, because it should state second *in Netherlands*. This question should be scrapped..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(341, 25, 2)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getQuestionAt(67425)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Title**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Energy\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Paragraph**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thermodynamics divides energy transformation into two kinds: reversible processes and irreversible processes. An irreversible process is one in which energy is dissipated (spread) into empty energy states available in a volume, from which it cannot be recovered into more concentrated forms (fewer quantum states), without degradation of even more energy. A reversible process is one in which this sort of dissipation does not happen. For example, conversion of energy from one type of potential field to another, is reversible, as in the pendulum system described above. In processes where heat is generated, quantum states of lower energy, present as possible excitations in fields between atoms, act as a reservoir for part of the energy, from which it cannot be recovered, in order to be converted with 100% efficiency into other forms of energy. In this case, the energy must partly stay as heat, and cannot be completely recovered as usable energy, except at the price of an increase in some other kind of heat-like increase in disorder in quantum states, in the universe (such as an expansion of matter, or a randomisation in a crystal).\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Question**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A reversible process is one in which this does not happen.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answer**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "406\n",
      "dissipation\n"
     ]
    }
   ],
   "source": [
    "titleId = 341\n",
    "paragraphId = 25 \n",
    "questionId = 2\n",
    "\n",
    "showQuestion(titleId, paragraphId, questionId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is, basically, just the question I expect to generate. The answer is removed and the sentence is descriptive enough to fill in the missing word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The assumption that the **question is mostly consisted of words from the sentence the answer is in** seems correct.\n",
    "\n",
    "There are some obvious differences like:\n",
    "- **Question-like words** are added - who, why, when...\n",
    "- **Synonyms** are used instead of the words used in the sentence\n",
    "- Changing the sentence to a question also changes the **tense** of the word.\n",
    "- In long sentences, only a **part of the sentence is used**. Like if the sentence is separated with commas, the comma actually divides two logical statements.\n",
    "\n",
    "I also managed to find some outliers which turned out to be not-so-well asked questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Couple of ideas to explore:\n",
    "- Are all the answers phrases from the text\n",
    "- The type of the answers - number, dates, names, locations, similarity to the title\n",
    "- Part of speech - verb, noun\n",
    "- Answer lenght in words\n",
    "- Words around the answer.\n",
    "- Answer location in the sentence - First word, last word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answers contained in the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Answers in text**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98169\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**Answers not in text**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "answersInText = 0\n",
    "answersNotInText = 0\n",
    "\n",
    "for titleId in range(len(df['data'])):\n",
    "     for paragraphId in range(len(df['data'][titleId]['paragraphs'])):\n",
    "        paragraph = df['data'][titleId]['paragraphs'][paragraphId]['context']\n",
    "        for questionId in range(len(df['data'][titleId]['paragraphs'][paragraphId]['qas'])):\n",
    "            answer = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['text']\n",
    "            if (answer in paragraph):\n",
    "                answersInText += 1\n",
    "            else:\n",
    "                answersNotInText += 1\n",
    "                \n",
    "printBold('Answers in text')\n",
    "print(answersInText)\n",
    "printBold('Answers not in text')\n",
    "print(answersNotInText)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the answers are phrases from the text. Seems like that has been a requirement from the start, since the answers also have an index indicating their start location in the paragraph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers = []\n",
    "sentences = []\n",
    "\n",
    "for titleId in range(len(df['data'])):\n",
    "    \n",
    "     for paragraphId in range(len(df['data'][titleId]['paragraphs'])):\n",
    "        paragraph = df['data'][titleId]['paragraphs'][paragraphId]['context']\n",
    "        \n",
    "        for questionId in range(len(df['data'][titleId]['paragraphs'][paragraphId]['qas'])):\n",
    "            answer = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['text']\n",
    "            answerStart = df['data'][titleId]['paragraphs'][paragraphId]['qas'][questionId]['answers'][0]['answer_start']\n",
    "            \n",
    "            sentence = extractSentence(paragraph, answerStart)\n",
    "            \n",
    "            answers.append(answer)\n",
    "            sentences.append(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Saint Bernadette Soubirous</td>\n",
       "      <td>It is a replica of the grotto at Lourdes, Fran...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a copper statue of Christ</td>\n",
       "      <td>Immediately in front of the Main Building and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the Main Building</td>\n",
       "      <td>Next to the Main Building is the Basilica of t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a Marian place of prayer and reflection</td>\n",
       "      <td>Immediately behind the basilica is the Grotto,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a golden statue of the Virgin Mary</td>\n",
       "      <td>Atop the Main Building's gold dome is a golden...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    answer  \\\n",
       "0               Saint Bernadette Soubirous   \n",
       "1                a copper statue of Christ   \n",
       "2                        the Main Building   \n",
       "3  a Marian place of prayer and reflection   \n",
       "4       a golden statue of the Virgin Mary   \n",
       "\n",
       "                                            sentence  \n",
       "0  It is a replica of the grotto at Lourdes, Fran...  \n",
       "1  Immediately in front of the Main Building and ...  \n",
       "2  Next to the Main Building is the Basilica of t...  \n",
       "3  Immediately behind the basilica is the Grotto,...  \n",
       "4  Atop the Main Building's gold dome is a golden...  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answerTextsDf = pd.DataFrame(answers, columns=['answer'])\n",
    "sentenceDf = pd.DataFrame(sentences, columns=['sentence'])\n",
    "\n",
    "answersDf = pd.concat([answerTextsDf, sentenceDf], axis=1)\n",
    "answersDf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answer word lenght "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordCount = []\n",
    "\n",
    "for i in range(len(answersDf)):\n",
    "    wordCount.append(len(tokenize.word_tokenize(answersDf.iloc[i]['answer'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "answersDf = pd.concat([answersDf, pd.DataFrame(wordCount, columns=['wordCount'])], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    98169.000000\n",
       "mean         3.354511\n",
       "std          3.731074\n",
       "min          1.000000\n",
       "25%          1.000000\n",
       "50%          2.000000\n",
       "75%          4.000000\n",
       "max         46.000000\n",
       "Name: wordCount, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['wordCount'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     32161\n",
       "2     25233\n",
       "3     14350\n",
       "4      7557\n",
       "5      4654\n",
       "6      3050\n",
       "7      2222\n",
       "8      1676\n",
       "9      1206\n",
       "10      974\n",
       "11      755\n",
       "12      653\n",
       "13      565\n",
       "14      462\n",
       "15      406\n",
       "16      313\n",
       "18      275\n",
       "17      269\n",
       "19      243\n",
       "20      191\n",
       "21      182\n",
       "23      138\n",
       "22      132\n",
       "25      120\n",
       "24      101\n",
       "26       78\n",
       "28       58\n",
       "27       57\n",
       "29       29\n",
       "30       18\n",
       "31       12\n",
       "32       11\n",
       "33        6\n",
       "38        2\n",
       "34        2\n",
       "35        2\n",
       "36        2\n",
       "37        2\n",
       "42        1\n",
       "46        1\n",
       "Name: wordCount, dtype: int64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['wordCount'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 1/3 of of the answers are single words. And about 2/3 are up to 3 words. Let's get an overview of the groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>52642</th>\n",
       "      <td>mul</td>\n",
       "      <td>For example, the name for the hanja 水 is 물 수 (...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79457</th>\n",
       "      <td>11,000–16,000</td>\n",
       "      <td>The total Iranian casualties in the war were e...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88678</th>\n",
       "      <td>Saracens</td>\n",
       "      <td>From these bases, the Normans eventually captu...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35390</th>\n",
       "      <td>microphone</td>\n",
       "      <td>The second controller lacked the START and SEL...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34469</th>\n",
       "      <td>rarely</td>\n",
       "      <td>Since Elizabeth rarely gives interviews, littl...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57333</th>\n",
       "      <td>1991</td>\n",
       "      <td>By the late 1980s, digital media, in the form ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10684</th>\n",
       "      <td>ZigBee</td>\n",
       "      <td>Many newer control systems are using wireless ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43080</th>\n",
       "      <td>1990s</td>\n",
       "      <td>Intergender singles bouts were first fought on...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43755</th>\n",
       "      <td>1870</td>\n",
       "      <td>In 1870, after France attacked Prussia, Prussi...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65525</th>\n",
       "      <td>Champs-Élysées</td>\n",
       "      <td>As of 2013 the City of Paris had 1,570 hotels ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               answer                                           sentence  \\\n",
       "52642             mul  For example, the name for the hanja 水 is 물 수 (...   \n",
       "79457   11,000–16,000  The total Iranian casualties in the war were e...   \n",
       "88678        Saracens  From these bases, the Normans eventually captu...   \n",
       "35390      microphone  The second controller lacked the START and SEL...   \n",
       "34469          rarely  Since Elizabeth rarely gives interviews, littl...   \n",
       "57333            1991  By the late 1980s, digital media, in the form ...   \n",
       "10684          ZigBee  Many newer control systems are using wireless ...   \n",
       "43080           1990s  Intergender singles bouts were first fought on...   \n",
       "43755            1870  In 1870, after France attacked Prussia, Prussi...   \n",
       "65525  Champs-Élysées  As of 2013 the City of Paris had 1,570 hotels ...   \n",
       "\n",
       "       wordCount  \n",
       "52642          1  \n",
       "79457          1  \n",
       "88678          1  \n",
       "35390          1  \n",
       "34469          1  \n",
       "57333          1  \n",
       "10684          1  \n",
       "43080          1  \n",
       "43755          1  \n",
       "65525          1  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 1].sample(10, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There seems to be a lot of years and some names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two word answers seem to be dominated by names. There are also a lot of answers where one of the words isn't useful. Some could easily be removed like *a* and *the*. *six years* and *tree times* could also be turned to just 6 and 3. The *13.3%* seems to be just misplaced. Not sure if it's because of the *\".\"* or the *\"%\"*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31777</th>\n",
       "      <td>six years</td>\n",
       "      <td>A peace agreement was signed in which John ret...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4799</th>\n",
       "      <td>Notre Dame</td>\n",
       "      <td>In 2006, Lee was awarded an honorary doctorate...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21766</th>\n",
       "      <td>gamma-aminobutyric acid</td>\n",
       "      <td>The two neurotransmitters that are used most w...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28267</th>\n",
       "      <td>Thomas Aquinas</td>\n",
       "      <td>During the Middle Ages, the Aristotelian view ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7152</th>\n",
       "      <td>The Beatles</td>\n",
       "      <td>The single, \"A Moment Like This\", went on to b...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26176</th>\n",
       "      <td>migratory species</td>\n",
       "      <td>The state is also a host to a large population...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33975</th>\n",
       "      <td>over five</td>\n",
       "      <td>For example, over five columns of text were de...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4851</th>\n",
       "      <td>Mockingbird groupies</td>\n",
       "      <td>Local residents call them \"Mockingbird groupie...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85540</th>\n",
       "      <td>Alan Rogerson</td>\n",
       "      <td>Former members Heather and Gary Botting compar...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77579</th>\n",
       "      <td>Sheffield United</td>\n",
       "      <td>The first ever Premier League goal was scored ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33022</th>\n",
       "      <td>suppressive fire</td>\n",
       "      <td>Many units are supplemented with a variety of ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19486</th>\n",
       "      <td>political boundaries</td>\n",
       "      <td>This claim also cannot be used to invalidate t...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92917</th>\n",
       "      <td>20 minutes</td>\n",
       "      <td>It is connected to the city via the Metro Ligh...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56025</th>\n",
       "      <td>Ashe County</td>\n",
       "      <td>Over the last decade, North Carolina has becom...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28701</th>\n",
       "      <td>Brian Labone</td>\n",
       "      <td>The late centre half and former captain Brian ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5075</th>\n",
       "      <td>learning investments</td>\n",
       "      <td>Hence the additional costs of the incentives f...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22254</th>\n",
       "      <td>23.02%</td>\n",
       "      <td>According to surveys conducted in 2007 and 200...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7646</th>\n",
       "      <td>Jennifer Hudson</td>\n",
       "      <td>Other alumni have gone on to work in televisio...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54428</th>\n",
       "      <td>every continent</td>\n",
       "      <td>Glaciers are present on every continent and ap...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29432</th>\n",
       "      <td>21 March</td>\n",
       "      <td>The Church of Alexandria celebrated Easter on ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        answer  \\\n",
       "31777                six years   \n",
       "4799                Notre Dame   \n",
       "21766  gamma-aminobutyric acid   \n",
       "28267           Thomas Aquinas   \n",
       "7152               The Beatles   \n",
       "26176        migratory species   \n",
       "33975                over five   \n",
       "4851      Mockingbird groupies   \n",
       "85540            Alan Rogerson   \n",
       "77579         Sheffield United   \n",
       "33022         suppressive fire   \n",
       "19486     political boundaries   \n",
       "92917               20 minutes   \n",
       "56025              Ashe County   \n",
       "28701             Brian Labone   \n",
       "5075      learning investments   \n",
       "22254                   23.02%   \n",
       "7646           Jennifer Hudson   \n",
       "54428          every continent   \n",
       "29432                 21 March   \n",
       "\n",
       "                                                sentence  wordCount  \n",
       "31777  A peace agreement was signed in which John ret...          2  \n",
       "4799   In 2006, Lee was awarded an honorary doctorate...          2  \n",
       "21766  The two neurotransmitters that are used most w...          2  \n",
       "28267  During the Middle Ages, the Aristotelian view ...          2  \n",
       "7152   The single, \"A Moment Like This\", went on to b...          2  \n",
       "26176  The state is also a host to a large population...          2  \n",
       "33975  For example, over five columns of text were de...          2  \n",
       "4851   Local residents call them \"Mockingbird groupie...          2  \n",
       "85540  Former members Heather and Gary Botting compar...          2  \n",
       "77579  The first ever Premier League goal was scored ...          2  \n",
       "33022  Many units are supplemented with a variety of ...          2  \n",
       "19486  This claim also cannot be used to invalidate t...          2  \n",
       "92917  It is connected to the city via the Metro Ligh...          2  \n",
       "56025  Over the last decade, North Carolina has becom...          2  \n",
       "28701  The late centre half and former captain Brian ...          2  \n",
       "5075   Hence the additional costs of the incentives f...          2  \n",
       "22254  According to surveys conducted in 2007 and 200...          2  \n",
       "7646   Other alumni have gone on to work in televisio...          2  \n",
       "54428  Glaciers are present on every continent and ap...          2  \n",
       "29432  The Church of Alexandria celebrated Easter on ...          2  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 2].sample(n=20, random_state=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two word answers seem to be dominated by names. There are also a lot of answers where one of the words isn't useful. Some could easily be removed like *a*, *in* and *the*. *six years* could be turned to just 6. The *23.02%* seems to be just misplaced. Not sure if it's because of the *\".\"* or the *\"%\"*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>49157</th>\n",
       "      <td>Vasco da Gama</td>\n",
       "      <td>Portugal had during the 15th century – particu...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28486</th>\n",
       "      <td>Copa del Generalísimo</td>\n",
       "      <td>The 1960s saw the emergence of Josep Maria Fus...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91654</th>\n",
       "      <td>magnetic tape shortage</td>\n",
       "      <td>During the following years, a magnetic tape sh...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95828</th>\n",
       "      <td>fear of betrayal</td>\n",
       "      <td>In 1354, when Toghtogha led a large army to cr...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61090</th>\n",
       "      <td>Arab Umayyad Caliphate</td>\n",
       "      <td>After conquering Persia, the Arab Umayyad Cali...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92998</th>\n",
       "      <td>keyed Northumbrian smallpipes</td>\n",
       "      <td>John Dunn, inventor of keyed Northumbrian smal...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66068</th>\n",
       "      <td>The Weather Company</td>\n",
       "      <td>On October 28, 2015, IBM announced its acquisi...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24543</th>\n",
       "      <td>10 February 1931</td>\n",
       "      <td>The city that was later dubbed \"Lutyens' Delhi...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50145</th>\n",
       "      <td>political and moral</td>\n",
       "      <td>He is without parallel in any age, excepting p...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84203</th>\n",
       "      <td>the Roku player</td>\n",
       "      <td>Google made YouTube available on the Roku play...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72640</th>\n",
       "      <td>1792 and 1793</td>\n",
       "      <td>The guillotines used during the Reign of Terro...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78086</th>\n",
       "      <td>power to veto</td>\n",
       "      <td>This made his person sacrosanct, gave him the ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85670</th>\n",
       "      <td>George C. Marshall</td>\n",
       "      <td>Next, he was appointed Assistant Chief of Staf...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10430</th>\n",
       "      <td>Pius V.</td>\n",
       "      <td>The use of the title was reserved for the card...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49201</th>\n",
       "      <td>Frederick the Wise</td>\n",
       "      <td>When he refused, he was placed under the ban o...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8290</th>\n",
       "      <td>Reporters Without Borders</td>\n",
       "      <td>Reporters Without Borders organised several sy...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82432</th>\n",
       "      <td>mythical chullumpi bird</td>\n",
       "      <td>The mythical chullumpi bird is said to mark th...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44100</th>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>New Haven is the birthplace of former presiden...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93328</th>\n",
       "      <td>president and CEO</td>\n",
       "      <td>Noble subsequently acquired the rights to the ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61195</th>\n",
       "      <td>East India Company</td>\n",
       "      <td>This led to the Battle of Plassey on 23 June 1...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              answer  \\\n",
       "49157                  Vasco da Gama   \n",
       "28486          Copa del Generalísimo   \n",
       "91654         magnetic tape shortage   \n",
       "95828               fear of betrayal   \n",
       "61090         Arab Umayyad Caliphate   \n",
       "92998  keyed Northumbrian smallpipes   \n",
       "66068            The Weather Company   \n",
       "24543               10 February 1931   \n",
       "50145            political and moral   \n",
       "84203                the Roku player   \n",
       "72640                  1792 and 1793   \n",
       "78086                  power to veto   \n",
       "85670             George C. Marshall   \n",
       "10430                        Pius V.   \n",
       "49201             Frederick the Wise   \n",
       "8290       Reporters Without Borders   \n",
       "82432        mythical chullumpi bird   \n",
       "44100                 George W. Bush   \n",
       "93328              president and CEO   \n",
       "61195             East India Company   \n",
       "\n",
       "                                                sentence  wordCount  \n",
       "49157  Portugal had during the 15th century – particu...          3  \n",
       "28486  The 1960s saw the emergence of Josep Maria Fus...          3  \n",
       "91654  During the following years, a magnetic tape sh...          3  \n",
       "95828  In 1354, when Toghtogha led a large army to cr...          3  \n",
       "61090  After conquering Persia, the Arab Umayyad Cali...          3  \n",
       "92998  John Dunn, inventor of keyed Northumbrian smal...          3  \n",
       "66068  On October 28, 2015, IBM announced its acquisi...          3  \n",
       "24543  The city that was later dubbed \"Lutyens' Delhi...          3  \n",
       "50145  He is without parallel in any age, excepting p...          3  \n",
       "84203  Google made YouTube available on the Roku play...          3  \n",
       "72640  The guillotines used during the Reign of Terro...          3  \n",
       "78086  This made his person sacrosanct, gave him the ...          3  \n",
       "85670  Next, he was appointed Assistant Chief of Staf...          3  \n",
       "10430  The use of the title was reserved for the card...          3  \n",
       "49201  When he refused, he was placed under the ban o...          3  \n",
       "8290   Reporters Without Borders organised several sy...          3  \n",
       "82432  The mythical chullumpi bird is said to mark th...          3  \n",
       "44100  New Haven is the birthplace of former presiden...          3  \n",
       "93328  Noble subsequently acquired the rights to the ...          3  \n",
       "61195  This led to the Battle of Plassey on 23 June 1...          3  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 3].sample(n=20, random_state=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again names, more institution names as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>85209</th>\n",
       "      <td>conduct surveys of party colleagues</td>\n",
       "      <td>For instance, to keep their party colleagues \"...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20338</th>\n",
       "      <td>Robert Bideleux and Ian Jeffries</td>\n",
       "      <td>Significant legislative changes in the status ...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22367</th>\n",
       "      <td>in excess of £3.3 billion</td>\n",
       "      <td>The total annual cost to support the defence e...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64526</th>\n",
       "      <td>Koninklijk Conservatorium Artesis Hogeschool A...</td>\n",
       "      <td>She is now also professor mandolin at the musi...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97009</th>\n",
       "      <td>end of World War I</td>\n",
       "      <td>At the end of World War I, the Rhineland was s...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54390</th>\n",
       "      <td>partly cold-based and partly warm-based</td>\n",
       "      <td>Glaciers which are partly cold-based and partl...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90084</th>\n",
       "      <td>body and blood of Christ</td>\n",
       "      <td>Luther insisted on the Real Presence of the bo...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39942</th>\n",
       "      <td>the eastern waterfront in Buceo</td>\n",
       "      <td>The Museo Naval, is located on the eastern wat...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95598</th>\n",
       "      <td>School of Social Service Administration</td>\n",
       "      <td>In 1955, Eero Saarinen was contracted to devel...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58652</th>\n",
       "      <td>protruded from the road surface</td>\n",
       "      <td>However, the company ceased trading in 1875 af...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56345</th>\n",
       "      <td>scientific naturalism over natural theology</td>\n",
       "      <td>Huxley wanted science to be secular, without r...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25781</th>\n",
       "      <td>Kraftwerk, Art of Noise</td>\n",
       "      <td>This sound, also influenced by European electr...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38132</th>\n",
       "      <td>a great-great grandson of Jacob</td>\n",
       "      <td>Jacob and his sons had lived in Canaan but wer...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41885</th>\n",
       "      <td>off Australia's northwestern coast</td>\n",
       "      <td>On 20 May 2011, Royal Dutch Shell's final inve...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64614</th>\n",
       "      <td>Chris Thile of California is</td>\n",
       "      <td>Chris Thile of California is a well-known play...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89455</th>\n",
       "      <td>in less than quadratic time</td>\n",
       "      <td>Similarly, algorithms can solve the NP-complet...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20808</th>\n",
       "      <td>between 1000 to 1500 BC</td>\n",
       "      <td>Celtic tribes settled in Switzerland between 1...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16151</th>\n",
       "      <td>executive director of football operations</td>\n",
       "      <td>Foster appointed legendary Darrel \"Mouse\" Davi...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55832</th>\n",
       "      <td>tobacco, cotton and agriculture</td>\n",
       "      <td>Impoverished by the Civil War, the state conti...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37697</th>\n",
       "      <td>large tumour on her liver</td>\n",
       "      <td>When Lady Flora died in July, the post-mortem ...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  answer  \\\n",
       "85209                conduct surveys of party colleagues   \n",
       "20338                   Robert Bideleux and Ian Jeffries   \n",
       "22367                          in excess of £3.3 billion   \n",
       "64526  Koninklijk Conservatorium Artesis Hogeschool A...   \n",
       "97009                                 end of World War I   \n",
       "54390            partly cold-based and partly warm-based   \n",
       "90084                           body and blood of Christ   \n",
       "39942                    the eastern waterfront in Buceo   \n",
       "95598            School of Social Service Administration   \n",
       "58652                    protruded from the road surface   \n",
       "56345        scientific naturalism over natural theology   \n",
       "25781                            Kraftwerk, Art of Noise   \n",
       "38132                    a great-great grandson of Jacob   \n",
       "41885                 off Australia's northwestern coast   \n",
       "64614                       Chris Thile of California is   \n",
       "89455                        in less than quadratic time   \n",
       "20808                            between 1000 to 1500 BC   \n",
       "16151          executive director of football operations   \n",
       "55832                    tobacco, cotton and agriculture   \n",
       "37697                          large tumour on her liver   \n",
       "\n",
       "                                                sentence  wordCount  \n",
       "85209  For instance, to keep their party colleagues \"...          5  \n",
       "20338  Significant legislative changes in the status ...          5  \n",
       "22367  The total annual cost to support the defence e...          5  \n",
       "64526  She is now also professor mandolin at the musi...          5  \n",
       "97009  At the end of World War I, the Rhineland was s...          5  \n",
       "54390  Glaciers which are partly cold-based and partl...          5  \n",
       "90084  Luther insisted on the Real Presence of the bo...          5  \n",
       "39942  The Museo Naval, is located on the eastern wat...          5  \n",
       "95598  In 1955, Eero Saarinen was contracted to devel...          5  \n",
       "58652  However, the company ceased trading in 1875 af...          5  \n",
       "56345  Huxley wanted science to be secular, without r...          5  \n",
       "25781  This sound, also influenced by European electr...          5  \n",
       "38132  Jacob and his sons had lived in Canaan but wer...          5  \n",
       "41885  On 20 May 2011, Royal Dutch Shell's final inve...          5  \n",
       "64614  Chris Thile of California is a well-known play...          5  \n",
       "89455  Similarly, algorithms can solve the NP-complet...          5  \n",
       "20808  Celtic tribes settled in Switzerland between 1...          5  \n",
       "16151  Foster appointed legendary Darrel \"Mouse\" Davi...          5  \n",
       "55832  Impoverished by the Civil War, the state conti...          5  \n",
       "37697  When Lady Flora died in July, the post-mortem ...          5  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 5].sample(n=20, random_state=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the words increase it seems harder to create deceptive incorrect answers. A viable option for some would to be mix the individual words like:\n",
    "\n",
    "*end of World War I\" -> start of World War 1, end of World War II, start of World War II, end of Balkans Wars*....\n",
    "\n",
    "*large tumour on her liver -> large tumor on her brain, large tumor on her lungs, large (some other medical term) on her liver*\n",
    "\n",
    "Though this would become more difficult because if use 2 generated words, they must also fit with each other as well as the original words.\n",
    "\n",
    "Some of the anwers look like logical phrases. For their generation I would argue that a text-summarization aproach would work. And with longer answers we could employ a **True/False** questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14715</th>\n",
       "      <td>to saturate broken (\"dangling\") bonds of amorp...</td>\n",
       "      <td>Hydrogen is employed to saturate broken (\"dang...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70704</th>\n",
       "      <td>Bullied for being a Bedouin, he was proud of h...</td>\n",
       "      <td>Bullied for being a Bedouin, he was proud of h...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39422</th>\n",
       "      <td>the Bill &amp; Melinda Gates Foundation Trust, whi...</td>\n",
       "      <td>In October 2006, the Bill &amp; Melinda Gates Foun...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21062</th>\n",
       "      <td>There are 64 possible codons (four possible nu...</td>\n",
       "      <td>:6 Additionally, a \"start codon\", and three \"s...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5882</th>\n",
       "      <td>Francesinha (Frenchie) from Porto, and bifanas...</td>\n",
       "      <td>Typical fast food dishes include the Francesin...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34718</th>\n",
       "      <td>into four summaries that look specifically at ...</td>\n",
       "      <td>However, results can be further simplified int...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26946</th>\n",
       "      <td>elected members and special office bearers suc...</td>\n",
       "      <td>The legislature consists of elected members an...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96100</th>\n",
       "      <td>support from China for a planned $2.5 billion ...</td>\n",
       "      <td>Kenyatta was \"[a]ccompanied by 60 Kenyan busin...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77789</th>\n",
       "      <td>On 26 December 1999, Chelsea became the first ...</td>\n",
       "      <td>On 26 December 1999, Chelsea became the first ...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97887</th>\n",
       "      <td>format of the congress and many specifics of t...</td>\n",
       "      <td>Nevertheless, the format of the congress and m...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  answer  \\\n",
       "14715  to saturate broken (\"dangling\") bonds of amorp...   \n",
       "70704  Bullied for being a Bedouin, he was proud of h...   \n",
       "39422  the Bill & Melinda Gates Foundation Trust, whi...   \n",
       "21062  There are 64 possible codons (four possible nu...   \n",
       "5882   Francesinha (Frenchie) from Porto, and bifanas...   \n",
       "34718  into four summaries that look specifically at ...   \n",
       "26946  elected members and special office bearers suc...   \n",
       "96100  support from China for a planned $2.5 billion ...   \n",
       "77789  On 26 December 1999, Chelsea became the first ...   \n",
       "97887  format of the congress and many specifics of t...   \n",
       "\n",
       "                                                sentence  wordCount  \n",
       "14715  Hydrogen is employed to saturate broken (\"dang...         20  \n",
       "70704  Bullied for being a Bedouin, he was proud of h...         20  \n",
       "39422  In October 2006, the Bill & Melinda Gates Foun...         20  \n",
       "21062  :6 Additionally, a \"start codon\", and three \"s...         20  \n",
       "5882   Typical fast food dishes include the Francesin...         20  \n",
       "34718  However, results can be further simplified int...         20  \n",
       "26946  The legislature consists of elected members an...         20  \n",
       "96100  Kenyatta was \"[a]ccompanied by 60 Kenyan busin...         20  \n",
       "77789  On 26 December 1999, Chelsea became the first ...         20  \n",
       "97887  Nevertheless, the format of the congress and m...         20  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 20].sample(n=10, random_state=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'On 26 December 1999, Chelsea became the first Premier League side to field an entirely foreign starting line-up,'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 20].sample(n=20, random_state=5).iloc[8]['answer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would argue that from this sentence could be created several questions with single word answers, like:\n",
    "- In what year? - *1999*\n",
    "- Which team? - *Chelsea*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our longest answer with 46 words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'that the sudden shift of a huge quantity of water into the region could have relaxed the tension between the two sides of the fault, allowing them to move apart, and could have increased the direct pressure on it, causing a violent rupture'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 46].iloc[0]['answer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*sudden shift of a huge quantity of water* seems like a good answer to the question *What could have relaxed the tension between the two sides?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hillary Clinton (2008), Howard Dean (2004), Gary Hart (1984 and 1988), Paul Tsongas (1992), Pat Robertson (1988) and Jerry Brown (1976, 1980, 1992).'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['wordCount'] == 42].iloc[0]['answer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second longest answer seems to be a sequence of correct answer, to something like *Who has been a presitend candidate*. This could be great for queastion with multiple correct answers as well as multiple incorrect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Spacy** turned out be a pretty great tool which could provide me with *NER (Named entity recognition), part of speech detection, word embeddings similarity* and some more functions which may or may be useful in my case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "from collections import Counter\n",
    "nlp = spacy.load('en_core_web_md')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Named entity recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('European', 'NORP'), ('Google', 'ORG'), ('$5.1 billion', 'MONEY'), ('Wednesday', 'DATE')]\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices')\n",
    "print([(X.text, X.label_) for X in doc.ents])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def NerForWord(text):\n",
    "    doc = nlp(text)\n",
    "    \n",
    "    entitiesFound = len(doc.ents)\n",
    "    \n",
    "    if (entitiesFound > 0):\n",
    "        #TODO - Could potentially find multiple entities in the text. We're returning only the first one.\n",
    "        return doc.ents[0].label_\n",
    "    else:\n",
    "        return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'GPE'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NerForWord('Portugal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Useful function for deciphering the tags. They really go deep into the grammatical types, most of which I haven't even heard  of until now. I suspect I'll have to group them up or not use some of the information at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'direct object'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy.explain(\"dobj\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the *spacy* tagging works on tokens (not necessarily single words, could be multiple words, e.g. names) it'll significatly ease my work if (for now) I work only with the answers which contain only 1 token. \n",
    "\n",
    "By my judgment, most of the multiple-token answers contain a single important token and a few words describing it. Or are multiple correct tokens separted by 'and' or ','. I could try to extract the important tokens, but I don't think it's worth it at this point.\n",
    "\n",
    "There are some great questions containing multi-token answers, but I think it's better If I limit myself to only single-token answers. That way I can work easier with word embedings and detect the tokens appropriate to be answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def isSingleToken(text):\n",
    "    doc = nlp(text)\n",
    "    \n",
    "    #The entire text is a single named entity \n",
    "    entitiesFound = len(doc.ents)\n",
    "    if(entitiesFound == 1 and doc.ents[0].text == text):\n",
    "        return True\n",
    "    \n",
    "    #The text is not an named entity, but is a single token\n",
    "    tokensFound = len(doc)\n",
    "    if (tokensFound == 1):\n",
    "        return True\n",
    "    \n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isSingleToken('George R. R. Martin')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how many of our answers we're gonna cut."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99%\n"
     ]
    }
   ],
   "source": [
    "singleTokenCount = 0\n",
    "\n",
    "sampleSize =  int(len(answersDf) / 10)\n",
    "for i in range(sampleSize):\n",
    "        \n",
    "    printPercentage(i, sampleSize)\n",
    "    \n",
    "    if (isSingleToken(answersDf.iloc[i]['answer'])):\n",
    "        singleTokenCount += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5843520782396088"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "singleTokenCount / sampleSize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On 10% of the data about 60% is retained. I expected worse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do some of the more interesting spacy tags - NER, POS, DEP, TAG, SHAPE...\n",
    "\n",
    "Better to provide the full text to spacy, because the token's tags are influenced by their relationship with the other words in the text. But we'll do that we do the feature engineering and also tag the non-answer words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "James james PROPN NNP compound Xxxxx True False 1 PERSON\n",
      "R. r. PROPN NNP compound X. False False 1 PERSON\n",
      "Scott scott PROPN NNP ROOT Xxxxx True False 1 PERSON\n",
      "Xxxxx X. Xxxxx\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('James R. Scott')\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "          token.shape_, token.is_alpha, token.is_stop, len(doc.ents), doc.ents[0].label_)\n",
    "    \n",
    "shape = doc[0].shape_\n",
    "for wordIndex in range(1, len(doc)):\n",
    "    shape += (' ' + doc[wordIndex].shape_)\n",
    "        \n",
    "print(shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Numerals that do not fall under another type'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy.explain('CARDINAL')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding the additional columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "answersDf['isSingleToken'] = False\n",
    "answersDf['NER'] = ''\n",
    "answersDf['POS'] = ''\n",
    "answersDf['TAG'] = ''\n",
    "answersDf['DEP'] = ''\n",
    "answersDf['shape'] = ''\n",
    "answersDf['isAlpha'] = False\n",
    "answersDf['isStop'] = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Saint Bernadette Soubirous</td>\n",
       "      <td>It is a replica of the grotto at Lourdes, Fran...</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a copper statue of Christ</td>\n",
       "      <td>Immediately in front of the Main Building and ...</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the Main Building</td>\n",
       "      <td>Next to the Main Building is the Basilica of t...</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a Marian place of prayer and reflection</td>\n",
       "      <td>Immediately behind the basilica is the Grotto,...</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a golden statue of the Virgin Mary</td>\n",
       "      <td>Atop the Main Building's gold dome is a golden...</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    answer  \\\n",
       "0               Saint Bernadette Soubirous   \n",
       "1                a copper statue of Christ   \n",
       "2                        the Main Building   \n",
       "3  a Marian place of prayer and reflection   \n",
       "4       a golden statue of the Virgin Mary   \n",
       "\n",
       "                                            sentence  wordCount  \\\n",
       "0  It is a replica of the grotto at Lourdes, Fran...          3   \n",
       "1  Immediately in front of the Main Building and ...          5   \n",
       "2  Next to the Main Building is the Basilica of t...          3   \n",
       "3  Immediately behind the basilica is the Grotto,...          7   \n",
       "4  Atop the Main Building's gold dome is a golden...          7   \n",
       "\n",
       "   isSingleToken NER POS TAG DEP shape  isAlpha  isStop  \n",
       "0          False                          False   False  \n",
       "1          False                          False   False  \n",
       "2          False                          False   False  \n",
       "3          False                          False   False  \n",
       "4          False                          False   False  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Populating the single-token answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99%\n"
     ]
    }
   ],
   "source": [
    "singleTokenCount = 0\n",
    "\n",
    "sampleSize = int(len(answersDf) / 10)\n",
    "\n",
    "for i in range(sampleSize):\n",
    "        \n",
    "    printPercentage(i, sampleSize)\n",
    "    \n",
    "    answer = answersDf.iloc[i]['answer']\n",
    "    if (isSingleToken(answer)):\n",
    "        answersDf.at[i, 'isSingleToken'] = True\n",
    "        \n",
    "        answersDf.at[i, 'NER'] = NerForWord(answer)\n",
    "        \n",
    "        #At this point I've called spacy's nlp method 3 times for the same words...\n",
    "        doc = nlp(answer)\n",
    "        \n",
    "        answersDf.at[i, 'POS'] = doc[0].pos_\n",
    "        answersDf.at[i, 'TAG'] = doc[0].tag_\n",
    "        answersDf.at[i, 'DEP'] = doc[0].dep_\n",
    "        answersDf.at[i, 'isAlpha'] = doc[0].is_alpha\n",
    "        answersDf.at[i, 'isStop'] = doc[0].is_stop\n",
    "        \n",
    "        shape = doc[0].shape_\n",
    "        for wordIndex in range(1, len(doc)):\n",
    "            shape += (' ' + doc[wordIndex].shape_)\n",
    "            \n",
    "        answersDf.at[i, 'shape'] = shape\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    98169\n",
       "Name: isStop, dtype: int64"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['isStop'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can safely not bother with stopwords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "               93503\n",
       "PERSON          1147\n",
       "CARDINAL        1069\n",
       "DATE             869\n",
       "ORG              691\n",
       "GPE              369\n",
       "PERCENT          156\n",
       "NORP              90\n",
       "MONEY             88\n",
       "LOC               48\n",
       "FAC               38\n",
       "QUANTITY          35\n",
       "ORDINAL           31\n",
       "WORK_OF_ART       14\n",
       "TIME               9\n",
       "EVENT              9\n",
       "PRODUCT            2\n",
       "LAW                1\n",
       "Name: NER, dtype: int64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['NER'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note that I've done the NER on only 10% of the dataset. So, 9350 out of 93503. Seems about 40% of the word have a NER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3774</th>\n",
       "      <td>UNICEF</td>\n",
       "      <td>On May 14, UNICEF reported that China formally...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>XXXX</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3210</th>\n",
       "      <td>Radiohead</td>\n",
       "      <td>The English band Radiohead also composed a son...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4107</th>\n",
       "      <td>Fort Tilden</td>\n",
       "      <td>Also in Queens, the park includes a significan...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2488</th>\n",
       "      <td>iTunes</td>\n",
       "      <td>Apple's iTunes software (and other alternative...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NNS</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xXxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9331</th>\n",
       "      <td>House Financial Services Committee</td>\n",
       "      <td>On September 10, 2003, the House Financial Ser...</td>\n",
       "      <td>4</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5769</th>\n",
       "      <td>TAP Portugal</td>\n",
       "      <td>The primary flag-carrier is TAP Portugal, alth...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>XXX Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6870</th>\n",
       "      <td>Soka Gakkai International</td>\n",
       "      <td>Soka Gakkai International (SGI) is a lay Buddh...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxx Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1921</th>\n",
       "      <td>Chopin</td>\n",
       "      <td>Possibly the first venture into fictional trea...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4471</th>\n",
       "      <td>Grand Central Station</td>\n",
       "      <td>The New York City Subway is also the busiest m...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9664</th>\n",
       "      <td>Congolese Labour Party</td>\n",
       "      <td>One year later, President Ngouabi proclaimed C...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>ORG</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  answer  \\\n",
       "3774                              UNICEF   \n",
       "3210                           Radiohead   \n",
       "4107                         Fort Tilden   \n",
       "2488                              iTunes   \n",
       "9331  House Financial Services Committee   \n",
       "5769                        TAP Portugal   \n",
       "6870           Soka Gakkai International   \n",
       "1921                              Chopin   \n",
       "4471               Grand Central Station   \n",
       "9664              Congolese Labour Party   \n",
       "\n",
       "                                               sentence  wordCount  \\\n",
       "3774  On May 14, UNICEF reported that China formally...          1   \n",
       "3210  The English band Radiohead also composed a son...          1   \n",
       "4107  Also in Queens, the park includes a significan...          2   \n",
       "2488  Apple's iTunes software (and other alternative...          1   \n",
       "9331  On September 10, 2003, the House Financial Ser...          4   \n",
       "5769  The primary flag-carrier is TAP Portugal, alth...          2   \n",
       "6870  Soka Gakkai International (SGI) is a lay Buddh...          3   \n",
       "1921  Possibly the first venture into fictional trea...          1   \n",
       "4471  The New York City Subway is also the busiest m...          3   \n",
       "9664  One year later, President Ngouabi proclaimed C...          3   \n",
       "\n",
       "      isSingleToken  NER    POS  TAG       DEP                    shape  \\\n",
       "3774           True  ORG  PROPN  NNP      ROOT                     XXXX   \n",
       "3210           True  ORG  PROPN  NNP      ROOT                    Xxxxx   \n",
       "4107           True  ORG  PROPN  NNP  compound               Xxxx Xxxxx   \n",
       "2488           True  ORG   NOUN  NNS      ROOT                   xXxxxx   \n",
       "9331           True  ORG  PROPN  NNP  compound  Xxxxx Xxxxx Xxxxx Xxxxx   \n",
       "5769           True  ORG  PROPN  NNP      ROOT                XXX Xxxxx   \n",
       "6870           True  ORG  PROPN  NNP  compound         Xxxx Xxxxx Xxxxx   \n",
       "1921           True  ORG  PROPN  NNP      ROOT                    Xxxxx   \n",
       "4471           True  ORG  PROPN  NNP  compound        Xxxxx Xxxxx Xxxxx   \n",
       "9664           True  ORG  PROPN  NNP  compound        Xxxxx Xxxxx Xxxxx   \n",
       "\n",
       "      isAlpha  isStop  \n",
       "3774     True   False  \n",
       "3210     True   False  \n",
       "4107     True   False  \n",
       "2488     True   False  \n",
       "9331     True   False  \n",
       "5769     True   False  \n",
       "6870     True   False  \n",
       "1921     True   False  \n",
       "4471     True   False  \n",
       "9664     True   False  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['NER'] == 'ORG'].sample(n=10, random_state=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    94056\n",
       "True      4113\n",
       "Name: isAlpha, dtype: int64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['isAlpha'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part of speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "         92433\n",
       "PROPN     2624\n",
       "NUM       1715\n",
       "NOUN       663\n",
       "ADJ        294\n",
       "DET        102\n",
       "VERB        87\n",
       "SYM         72\n",
       "ADP         61\n",
       "ADV         51\n",
       "X           45\n",
       "PUNCT       14\n",
       "INTJ         7\n",
       "PRON         1\n",
       "Name: POS, dtype: int64"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf['POS'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Nouns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The answers are dominated by nouns. Difference between a noun and a proper noun (PROPN) is that proper nouns are names of specific people, places, ideas... while common nouns are just non-specific (cat, woman, bottle...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8461</th>\n",
       "      <td>Stephen Smith</td>\n",
       "      <td>Foreign Minister Stephen Smith said Chinese of...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4695</th>\n",
       "      <td>Truman Capote</td>\n",
       "      <td>Lee modeled the character of Dill on her child...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>772</th>\n",
       "      <td>L.A. Reid</td>\n",
       "      <td>The ex-President of Def Jam L.A. Reid has desc...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>X.X. Xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8160</th>\n",
       "      <td>Panathinaiko Stadium</td>\n",
       "      <td>After being lit at the birthplace of the Olymp...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>FAC</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>441</th>\n",
       "      <td>Shakira</td>\n",
       "      <td>At the same time, B'Day was re-released with f...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    answer                                           sentence  \\\n",
       "8461         Stephen Smith  Foreign Minister Stephen Smith said Chinese of...   \n",
       "4695         Truman Capote  Lee modeled the character of Dill on her child...   \n",
       "772              L.A. Reid  The ex-President of Def Jam L.A. Reid has desc...   \n",
       "8160  Panathinaiko Stadium  After being lit at the birthplace of the Olymp...   \n",
       "441                Shakira  At the same time, B'Day was re-released with f...   \n",
       "\n",
       "      wordCount  isSingleToken     NER    POS  TAG       DEP        shape  \\\n",
       "8461          2           True  PERSON  PROPN  NNP  compound  Xxxxx Xxxxx   \n",
       "4695          2           True  PERSON  PROPN  NNP  compound  Xxxxx Xxxxx   \n",
       "772           2           True  PERSON  PROPN  NNP  compound    X.X. Xxxx   \n",
       "8160          2           True     FAC  PROPN  NNP  compound  Xxxxx Xxxxx   \n",
       "441           1           True  PERSON  PROPN  NNP      ROOT        Xxxxx   \n",
       "\n",
       "      isAlpha  isStop  \n",
       "8461     True   False  \n",
       "4695     True   False  \n",
       "772     False   False  \n",
       "8160     True   False  \n",
       "441      True   False  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['POS'] == 'PROPN'].sample(n=5, random_state=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>shoe</td>\n",
       "      <td>In 2005, Beyoncé teamed up with House of Brand...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NN</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6504</th>\n",
       "      <td>arūpajhānas</td>\n",
       "      <td>Rebirths in the Ārūpyadhātu (formless realms) ...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NNS</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6540</th>\n",
       "      <td>ignorance</td>\n",
       "      <td>Within the context of the four noble truths, t...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NN</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3932</th>\n",
       "      <td>capital</td>\n",
       "      <td>In 1785, the assembly of the Congress of the C...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NN</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2550</th>\n",
       "      <td>bass</td>\n",
       "      <td>The third-generation iPod had a weak bass resp...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NN</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           answer                                           sentence  \\\n",
       "977          shoe  In 2005, Beyoncé teamed up with House of Brand...   \n",
       "6504  arūpajhānas  Rebirths in the Ārūpyadhātu (formless realms) ...   \n",
       "6540    ignorance  Within the context of the four noble truths, t...   \n",
       "3932      capital  In 1785, the assembly of the Congress of the C...   \n",
       "2550         bass  The third-generation iPod had a weak bass resp...   \n",
       "\n",
       "      wordCount  isSingleToken NER   POS  TAG   DEP shape  isAlpha  isStop  \n",
       "977           1           True      NOUN   NN  ROOT  xxxx     True   False  \n",
       "6504          1           True      NOUN  NNS  ROOT  xxxx     True   False  \n",
       "6540          1           True      NOUN   NN  ROOT  xxxx     True   False  \n",
       "3932          1           True      NOUN   NN  ROOT  xxxx     True   False  \n",
       "2550          1           True      NOUN   NN  ROOT  xxxx     True   False  "
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['POS'] == 'NOUN'].sample(n=5, random_state=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Numerals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second most prominent category is NUM. It's pretty much years and other numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6875</th>\n",
       "      <td>360 million</td>\n",
       "      <td>According to a demographic analysis reported b...</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>compound</td>\n",
       "      <td>ddd xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4065</th>\n",
       "      <td>1930</td>\n",
       "      <td>The character of New York's large residential ...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>DATE</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9597</th>\n",
       "      <td>1949</td>\n",
       "      <td>In 1946, for example, Walworth School was one ...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>DATE</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3275</th>\n",
       "      <td>75</td>\n",
       "      <td>And while Lea Seydoux doesn’t leave a huge imp...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2468</th>\n",
       "      <td>1720</td>\n",
       "      <td>When the Dzungar Mongols attempted to spread t...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2514</th>\n",
       "      <td>2007</td>\n",
       "      <td>In 2007, Apple modified the iPod interface aga...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>DATE</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1784</th>\n",
       "      <td>seven</td>\n",
       "      <td>They performed together on seven occasions bet...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1136</th>\n",
       "      <td>1,032,949</td>\n",
       "      <td>The United States Census Bureau estimates that...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>d,ddd,ddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8336</th>\n",
       "      <td>three</td>\n",
       "      <td>On April 7, 2008, two days prior to the actual...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7106</th>\n",
       "      <td>6,000</td>\n",
       "      <td>The finale for season two took place at the Gi...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>d,ddd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           answer                                           sentence  \\\n",
       "6875  360 million  According to a demographic analysis reported b...   \n",
       "4065         1930  The character of New York's large residential ...   \n",
       "9597         1949  In 1946, for example, Walworth School was one ...   \n",
       "3275           75  And while Lea Seydoux doesn’t leave a huge imp...   \n",
       "2468         1720  When the Dzungar Mongols attempted to spread t...   \n",
       "2514         2007  In 2007, Apple modified the iPod interface aga...   \n",
       "1784        seven  They performed together on seven occasions bet...   \n",
       "1136    1,032,949  The United States Census Bureau estimates that...   \n",
       "8336        three  On April 7, 2008, two days prior to the actual...   \n",
       "7106        6,000  The finale for season two took place at the Gi...   \n",
       "\n",
       "      wordCount  isSingleToken       NER  POS TAG       DEP      shape  \\\n",
       "6875          2           True  CARDINAL  NUM  CD  compound   ddd xxxx   \n",
       "4065          1           True      DATE  NUM  CD      ROOT       dddd   \n",
       "9597          1           True      DATE  NUM  CD      ROOT       dddd   \n",
       "3275          1           True  CARDINAL  NUM  CD      ROOT         dd   \n",
       "2468          1           True  CARDINAL  NUM  CD      ROOT       dddd   \n",
       "2514          1           True      DATE  NUM  CD      ROOT       dddd   \n",
       "1784          1           True  CARDINAL  NUM  CD      ROOT       xxxx   \n",
       "1136          1           True  CARDINAL  NUM  CD      ROOT  d,ddd,ddd   \n",
       "8336          1           True  CARDINAL  NUM  CD      ROOT       xxxx   \n",
       "7106          1           True  CARDINAL  NUM  CD      ROOT      d,ddd   \n",
       "\n",
       "      isAlpha  isStop  \n",
       "6875    False   False  \n",
       "4065    False   False  \n",
       "9597    False   False  \n",
       "3275    False   False  \n",
       "2468    False   False  \n",
       "2514    False   False  \n",
       "1784     True   False  \n",
       "1136    False   False  \n",
       "8336     True   False  \n",
       "7106    False   False  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[answersDf['POS'] == 'NUM'].sample(n=10, random_state=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adjectives and Verbs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Didn't really expect much of those, but they seem like adequate answers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3001</th>\n",
       "      <td>4th</td>\n",
       "      <td>IGN ranked the game as the 4th-best Wii game.</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>ORDINAL</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>dxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5033</th>\n",
       "      <td>thermal</td>\n",
       "      <td>Phase change materials such as paraffin wax an...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5626</th>\n",
       "      <td>subtropical</td>\n",
       "      <td>Both the archipelagos of the Azores and Madeir...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5308</th>\n",
       "      <td>Linguistic</td>\n",
       "      <td>Linguistic anthropology (also called anthropol...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7924</th>\n",
       "      <td>apex</td>\n",
       "      <td>Although large wild dogs, like wolves, are ape...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           answer                                           sentence  \\\n",
       "3001          4th      IGN ranked the game as the 4th-best Wii game.   \n",
       "5033      thermal  Phase change materials such as paraffin wax an...   \n",
       "5626  subtropical  Both the archipelagos of the Azores and Madeir...   \n",
       "5308   Linguistic  Linguistic anthropology (also called anthropol...   \n",
       "7924         apex  Although large wild dogs, like wolves, are ape...   \n",
       "\n",
       "      wordCount  isSingleToken      NER  POS TAG   DEP  shape  isAlpha  isStop  \n",
       "3001          1           True  ORDINAL  ADJ  JJ  ROOT    dxx    False   False  \n",
       "5033          1           True           ADJ  JJ  ROOT   xxxx     True   False  \n",
       "5626          1           True           ADJ  JJ  ROOT   xxxx     True   False  \n",
       "5308          1           True           ADJ  JJ  ROOT  Xxxxx     True   False  \n",
       "7924          1           True           ADJ  JJ  ROOT   xxxx     True   False  "
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[(answersDf['POS'] == 'ADJ') & (answersDf['wordCount'] == 1)].sample(n=5, random_state=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9454</th>\n",
       "      <td>leeward</td>\n",
       "      <td>They are categorized and divided into two grou...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>VERB</td>\n",
       "      <td>VB</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3894</th>\n",
       "      <td>Run</td>\n",
       "      <td>At the end of the Second Anglo-Dutch War, the ...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>VERB</td>\n",
       "      <td>VB</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9456</th>\n",
       "      <td>windsurfing</td>\n",
       "      <td>The windward beaches are popular for windsurfing.</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBG</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2858</th>\n",
       "      <td>Link</td>\n",
       "      <td>Enemies react to defeated companions and to ar...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>VERB</td>\n",
       "      <td>VB</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>Xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2856</th>\n",
       "      <td>react</td>\n",
       "      <td>Enemies react to defeated companions and to ar...</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td></td>\n",
       "      <td>VERB</td>\n",
       "      <td>VB</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           answer                                           sentence  \\\n",
       "9454      leeward  They are categorized and divided into two grou...   \n",
       "3894          Run  At the end of the Second Anglo-Dutch War, the ...   \n",
       "9456  windsurfing  The windward beaches are popular for windsurfing.   \n",
       "2858         Link  Enemies react to defeated companions and to ar...   \n",
       "2856        react  Enemies react to defeated companions and to ar...   \n",
       "\n",
       "      wordCount  isSingleToken NER   POS  TAG   DEP shape  isAlpha  isStop  \n",
       "9454          1           True      VERB   VB  ROOT  xxxx     True   False  \n",
       "3894          1           True      VERB   VB  ROOT   Xxx     True   False  \n",
       "9456          1           True      VERB  VBG  ROOT  xxxx     True   False  \n",
       "2858          1           True      VERB   VB  ROOT  Xxxx     True   False  \n",
       "2856          1           True      VERB   VB  ROOT  xxxx     True   False  "
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[(answersDf['POS'] == 'VERB') & (answersDf['wordCount'] == 1)].sample(n=5, random_state=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Symbols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the symbols are multi word answers, with some dollar signs infront."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>sentence</th>\n",
       "      <th>wordCount</th>\n",
       "      <th>isSingleToken</th>\n",
       "      <th>NER</th>\n",
       "      <th>POS</th>\n",
       "      <th>TAG</th>\n",
       "      <th>DEP</th>\n",
       "      <th>shape</th>\n",
       "      <th>isAlpha</th>\n",
       "      <th>isStop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3698</th>\n",
       "      <td>US$208 million</td>\n",
       "      <td>Donations of the evening totalled 1.5 billion ...</td>\n",
       "      <td>4</td>\n",
       "      <td>True</td>\n",
       "      <td>MONEY</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>XX$ ddd xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9259</th>\n",
       "      <td>US$2.5 trillion</td>\n",
       "      <td>During the last quarter of 2008, these central...</td>\n",
       "      <td>4</td>\n",
       "      <td>True</td>\n",
       "      <td>MONEY</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>XX$ d.d xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9067</th>\n",
       "      <td>$70 trillion</td>\n",
       "      <td>In a Peabody Award winning program, NPR corres...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>MONEY</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>$ dd xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3248</th>\n",
       "      <td>$8.2 million</td>\n",
       "      <td>Spectre opened in Germany with $22.45 million ...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>MONEY</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>$ d.d xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3569</th>\n",
       "      <td>$457 million</td>\n",
       "      <td>On May 16 China stated it had also received $4...</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>MONEY</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>$ ddd xxxx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               answer                                           sentence  \\\n",
       "3698   US$208 million  Donations of the evening totalled 1.5 billion ...   \n",
       "9259  US$2.5 trillion  During the last quarter of 2008, these central...   \n",
       "9067     $70 trillion  In a Peabody Award winning program, NPR corres...   \n",
       "3248     $8.2 million  Spectre opened in Germany with $22.45 million ...   \n",
       "3569     $457 million  On May 16 China stated it had also received $4...   \n",
       "\n",
       "      wordCount  isSingleToken    NER  POS TAG       DEP         shape  \\\n",
       "3698          4           True  MONEY  SYM   $  quantmod  XX$ ddd xxxx   \n",
       "9259          4           True  MONEY  SYM   $  quantmod  XX$ d.d xxxx   \n",
       "9067          3           True  MONEY  SYM   $  quantmod     $ dd xxxx   \n",
       "3248          3           True  MONEY  SYM   $  quantmod    $ d.d xxxx   \n",
       "3569          3           True  MONEY  SYM   $  quantmod    $ ddd xxxx   \n",
       "\n",
       "      isAlpha  isStop  \n",
       "3698    False   False  \n",
       "9259    False   False  \n",
       "9067    False   False  \n",
       "3248    False   False  \n",
       "3569    False   False  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answersDf[(answersDf['POS'] == 'SYM')].sample(n=5, random_state=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answers in a bigger picture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the highlighted answers.\n",
    "\n",
    "I suspect:\n",
    "\n",
    "1. There are other (many more?) obviously good words for answers that were just not selected.\n",
    "2. There are some sentences that just don't contain answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def highlightAnswers(titleId, paragraphId):\n",
    "\n",
    "    paragraph = df['data'][titleId]['paragraphs'][paragraphId]['context']\n",
    "    \n",
    "    answers = df['data'][titleId]['paragraphs'][paragraphId]['qas']\n",
    "\n",
    "    #Get answer starts and answer length\n",
    "    answerPosition = {}\n",
    "    for answer in answers:\n",
    "        answerStart = answer['answers'][0]['answer_start']\n",
    "        answerLength = len(answer['answers'][0]['text'])\n",
    "\n",
    "        answerPosition[answerStart] = answerLength\n",
    "\n",
    "    #Bold answers\n",
    "    shiftStart = 0\n",
    "    highlightedText = ''\n",
    "    currentPlaceInText = 0\n",
    "    \n",
    "    #Append text between previous answer and current answer + bold sign + answer + bold sign\n",
    "    for answerStart in sorted(answerPosition.keys()):\n",
    "        highlightedText += paragraph[currentPlaceInText:answerStart]\n",
    "        highlightedText += '**'\n",
    "        highlightedText += paragraph[answerStart:answerStart + answerPosition[answerStart]]\n",
    "        highlightedText += '**'\n",
    "        \n",
    "        currentPlaceInText = answerStart + answerPosition[answerStart]\n",
    "    \n",
    "    #Append the remaining text after the last answer\n",
    "    highlightedText += paragraph[currentPlaceInText:len(paragraph)]\n",
    "\n",
    "    #Diplay the highlighted text\n",
    "    display(Markdown(highlightedText))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Located approximately 250 kilometres (**160** mi) east of Puerto Rico and the nearer Virgin Islands, St. Barthélemy lies immediately southeast of the islands of Saint Martin and Anguilla. It is one of **the Renaissance** Islands. St. Barthélemy is separated from Saint Martin by **the Saint-Barthélemy Channel**. It lies northeast of Saba and St Eustatius, and north of St Kitts. Some small **satellite islets** belong to St. Barthélemy including Île Chevreau (Île Bonhomme), Île Frégate, Île Toc Vers, Île Tortue and Gros Îlets (Îlots Syndare). A much bigger islet, Île Fourchue, lies on the north of the island, in the Saint-Barthélemy Channel. Other rocky islets which include Coco, the Roques (or **little Turtle rocks**), the Goat, and the Sugarloaf."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titleId = 24\n",
    "paragraphId = 0\n",
    "\n",
    "highlightAnswers(titleId, paragraphId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Inappropriate antibiotic treatment and overuse** of antibiotics have contributed to the emergence of antibiotic-resistant bacteria. **Self prescription** of antibiotics is an example of misuse. Many antibiotics are frequently prescribed to treat symptoms or diseases that do not respond to antibiotics or that are likely to resolve without treatment. Also, incorrect or suboptimal antibiotics are prescribed for certain bacterial infections. The **overuse of antibiotics**, like penicillin and erythromycin, has been associated with emerging antibiotic resistance since the 1950s. Widespread usage of antibiotics in hospitals has also been associated with increases in bacterial strains and species that no longer respond to treatment with the most common antibiotics."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titleId = 4\n",
    "paragraphId = 12\n",
    "\n",
    "highlightAnswers(titleId, paragraphId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "According to the **endurance running hypothesis**, long-distance running as in **persistence hunting**, a method still practiced by **some hunter-gatherer groups** in modern times, was likely the driving evolutionary force leading to the evolution of certain human characteristics. This hypothesis does not necessarily contradict the **scavenging hypothesis**: **both subsistence strategies** could have been in use – sequentially, alternating or even simultaneously."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titleId = 52\n",
    "paragraphId = 4\n",
    "\n",
    "highlightAnswers(titleId, paragraphId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The first commercially successful true engine, in that it could generate power and transmit it to a machine, was the **atmospheric engine**, invented by **Thomas Newcomen** around **1712**. It was an improvement over Savery's **steam pump**, using a piston as proposed by **Papin**. Newcomen's engine was relatively inefficient, and in most cases was used for pumping water. It worked by creating a partial vacuum by condensing steam under a piston within a cylinder. It was employed for draining mine workings at depths hitherto impossible, and also for providing a reusable water supply for driving waterwheels at factories sited away from a suitable \"head\". Water that had passed over the wheel was pumped back up into a storage reservoir above the wheel."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titleId = 453\n",
    "paragraphId = 1\n",
    "\n",
    "highlightAnswers(titleId, paragraphId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It definetely seems that there are a lot more words that could become good answers. But I'm optimistic I can extract the selected word's features even if I don't have all of the possible answer words.\n",
    "\n",
    "At first glance, it seems like the answers are spread troughout the entire text and there aren't as many sentences without an answers. Though a better experiment would be to just count the sentences without answers agaisnt the ones with. \n",
    "But I don't see a large enough benefit to do it (deadline aproaching)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Noun chunks\n",
    "Another neat thing spacy gives us is noun chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the school\n",
      "a Catholic character\n",
      "the Main Building's gold dome\n",
      "a golden statue\n",
      "the Virgin Mary\n",
      "front\n",
      "the Main Building\n",
      "it\n",
      "a copper statue\n",
      "Christ\n",
      "arms\n",
      "the legend\n",
      "Venite Ad Me Omnes\n",
      "the Main Building\n",
      "the Basilica\n",
      "the Sacred Heart\n",
      "the basilica\n",
      "the Grotto\n",
      "a Marian place\n",
      "prayer\n",
      "reflection\n",
      "It\n",
      "a replica\n",
      "the grotto\n",
      "Lourdes\n",
      "France\n",
      "the Virgin Mary\n",
      "Saint Bernadette Soubirous\n",
      "the end\n",
      "the main drive\n",
      "a direct line\n",
      "3 statues\n",
      "the Gold Dome\n",
      "a simple, modern stone statue\n",
      "Mary\n"
     ]
    }
   ],
   "source": [
    "text = df['data'][0]['paragraphs'][0]['context']\n",
    "doc = nlp(text)\n",
    "\n",
    "for noun_chunk in doc.noun_chunks:\n",
    "    print(noun_chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is **a golden statue of the Virgin Mary**. Immediately in front of the Main Building and facing it, is **a copper statue of Christ** with arms upraised with the legend \"Venite Ad Me Omnes\". Next to **the Main Building** is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, **a Marian place of prayer and reflection**. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to **Saint Bernadette Soubirous** in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titleId = 0\n",
    "paragraphId = 0\n",
    "\n",
    "highlightAnswers(titleId, paragraphId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first paragraph we have 2 of the answers entirely recognized as noun chunks:\n",
    "1. **the Main Building**\n",
    "2. **Saint Bernadette Soubirous**\n",
    "\n",
    "While the other 3 answers are partially cut:\n",
    "1. **a golden statue** of the Virgin Mary\n",
    "2. **a copper statue** of *Christ* \n",
    "3. **a Marian place** of *prayer* and *reflection*\n",
    "\n",
    "Though I would argue that all of the other noun chunks would make great answers.\n",
    "I could potentially use only noun chunks for the answers and sacrifice the verbs and adjectives. But the noun chunks are mostly multi-word tokens. That would pose a problem with my features:\n",
    "1. **Part of speech** - Coulnd't really do it on multiple words.\n",
    "2. **TF-IDF** - Would need to modify it by either getting the aggreate of the single words or scoring the entire noun chunk... or both.\n",
    "3. **Title similarity** - Aggregation of the single words.\n",
    "4. **Incorrect answers** - That would be tricky, because I would need to find similar words for each word in the chunk and mix and match with the other similar words... That is bound to produce some inadequte mixes. But it still may not be a bad thing if I rely on a final filtering by a human."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
